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Initiative 502 (1-502) legalized recreational marijuana for adults in Washington State. The law directs 
the Washington State Institute for Public Policy (WSIPP) to conduct a benefit-cost evaluation of the 
implementation of 1-502.^ State law also requires the Health Care Authority's Division of Behavioral 
Health and Recovery (DBHR)^ to expend substance abuse prevention funding derived from cannabis 
revenues on programs demonstrated to be effective. Specifically, the law requires at least 85% of 
programs funded by cannabis revenues to be evidence-based or research-based and up to 15% to 
be promising practices.^ 

In this report, we summarize the research evidence for a set of programs intended for the prevention 
or treatment of youth substance use. The programs reviewed include those nominated by DBHR as 
well as similar programs from WSIPP's current set of inventories that have been evaluated for 
cannabis outcomes.'^ We rate the level of evidence for each program using the same methods used 
in other WSIPP inventories, as described below. 

This inventory is not limited to effective programs; we report on all programs reviewed, whether or 
not we find evidence of effectiveness. It is important to note that a wide variety of outcomes may be 
examined for a given program. Our evidence ratings are based on all relevant outcomes reported in 
the research, so it is possible that a given program is effective in preventing or treating the use of 
some substances but not others. It is also possible that a program is effective for related outcomes 
such as crime or risky sexual behavior but not for substance use. In addition to the overall evidence 
rating for each program, we also denote which programs have demonstrated evidence of 
effectiveness for preventing or treating cannabis use. Complete detailed results with specific 
outcome effects for each program can be found on WSIPP's website.^ 

This inventory is a snapshot of the evidence at a point in time.® Ratings for a program may change as 
new research becomes available and refinements are made to the WSIPP benefit-cost model. 


^ RCW 69.50.550. 

^ Recently re-located from the Department of Social and Health Services to the Health Care Authority. 

^ RCW 69.50.540. 

Miller, M., Goodvin, R., Grice, J., Hoagland, C., & Westley, E. (2016). Updated Inventory of evidence-based and research-based practices: 
Prevention and intervention services for aduit behavioral health. (Doc. No. 16-09-4101). Olympia: Washington State Institute for Public 
Policy; Cramer, J., Bitney, K., 8i Wanner, P. (2018). Updated inventory of evidence- and research-based practices: Washington's K-12 Learning 
Assistance Program. (Doc. No. 18-06-2201). Olympia: Washington State Institute for Public Policy; and EBPI 8i WSIPP. (2018). Updated 
inventory of evidence-based, research-based, and promising practices: For prevention and intervention services for children and Juveniles in 
the child welfare. Juvenile Justice, and mental health systems. (Doc. No. E2SHB2536-9). Olympia: Washington State Institute for Public Policy. 
^ Washington State Institute for Public Policy. Benefit-cost results. Olympia, WA: Author. 

^ This inventory is an update of a previous inventory; the most recent prior version is Darnell, A., Goodvin, R., Lemon, M. 8i Miller, M. (2016). 
Preventing and treating youth marijuana use: An updated review of the evidence. (Doc. No. 16-12-3201). Olympia: Washington State 
Institute for Public Policy. 
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Creating the Youth Cannabis Inventory 


WSIPP's approach to creating the inventory is the same approach we use for legislatively 
directed inventories in other policy areas/ We first use a rigorous, three-step research approach 
to assess the evidence, economics, and risk for each program. Then, using information derived 
from the three-step approach, we classify all programs according to standard definitions. 
WSIPP's three-step approach is as follows: 

1) Identify what works (and what does not). For each program under consideration, we 
systematically review all rigorous research evidence and estimate the program's effect on all 
relevant outcomes. The evidence may indicate that a program worked (i.e., had a desirable 
effect on outcomes), caused harm (i.e., had an undesirable effect on outcomes), or had no 
detectable effect one way or the other. 

2) Assess the return on investment. Given the estimated effect of a program from Step 1, we 
estimate—in dollars and cents—how much the program would benefit people in 
Washington were it implemented and how much it would cost the taxpayers to achieve this 
result. We use WSIPP's benefit-cost model to develop standardized, comparable results for 
all programs that illustrate the expected return on investment. We present these results as 
net present values on a per-participant basis. We also consider how monetary benefits are 
distributed across program participants, taxpayers, and other people in society. 

3) Determine the risk of investment. We allow for uncertainty in our estimates by calculating 
the probability that a program will at least "break even" if critical factors—like the actual cost 
to implement the program and the precise effect on the program—are lower or higher than 
our estimates. 

We follow a set of standardized procedures (see Exhibit 1) for each of these steps. These 
standardized procedures support the rigor of our analyses and allow programs to be compared 
on an "apples-to-apples" basis. For full detail on WSIPP's methods, see WSIPP's Technical 
Documentation.® 


’ EBPI, & WSIPP. (2018). 

* Washington State Institute for Public Policy. (December 2018). Benefit-cost technical documentation. Olympia, WA: Author. 
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Exhibit 1 

WSIPP's Three-Step Approach 


Step 1: Identify what works (and what does not) 

We conduct a nneta-analysis—a quantitative review of the research literature—to deternnine if the 
weight of the research evidence indicates whether desired outcomes are achieved, on average. 

WSIPP follows several key protocols to ensure a rigorous analysis for each program examined. We: 

• Search for all studies on a topic —We systematically review the national and international 
research literature and consider all available studies on a program, regardless of their 
findings. That is, we do not "cherry pick" studies to include in our analysis. 

• Screen studies for quality —We only include rigorous studies in our analysis. We require that a 
study reasonably attempt to demonstrate causality using appropriate statistical techniques. 

For example, studies must include both treatment and comparison groups with an intent-to- 
treat analysis. Studies that do not meet our minimum standards are excluded from analysis. 

• Determine the average effect size —We use a formal set of statistical procedures to calculate 
an average effect size for each outcome, which indicates the expected magnitude of change 
caused by the program (e.g., tutoring by adults) for each outcome of interest (e.g., 
standardized test scores). 

Step 2: Assess the return on investment 

WSIPP has developed, and continues to refine, an economic model to provide internally consistent 
monetary valuations of the benefits and costs of each program on a per-participant basis. 

Benefits to individuals and society may stem from multiple sources. For example, a program that 
reduces the need for publicly funded substance use treatment services decreases taxpayer costs. If 
that program also improves participants' educational outcomes, it will increase their expected 
labor market earnings. Finally, if a program reduces crime, it will reduce expected costs to crime 
victims. 

We also estimate the cost required to implement an intervention. If the program is operating in 
Washington State, our preferred method is to obtain the service delivery and administrative costs 
from state or local agencies. When this approach is not possible, we estimate costs using the 
research literature, using estimates provided by program developers, or using a variety of sources 
to construct our own cost estimate. 

Step 3: Determine the risk of investment 

Any tabulation of benefits and costs involves a degree of uncertainty about the inputs used in the 
analysis, as well as the bottom-line estimates. An assessment of risk is expected in any investment 
analysis, whether in the private or public sector. 

To assess the riskiness of our conclusions, we look at thousands of different scenarios through a 
Monte Carlo simulation. In each scenario, we vary a number of key factors in our calculations (e.g., 
expected effect sizes, program costs) using estimates of error around each factor. The purpose of this 
analysis is to determine the probability that a particular program or policy will produce benefits that 
are equal to or greater than costs if the real-world conditions are different than our baseline 
assumptions. 
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Classifying Practices as Evidence-Based, Research-Based, or Promising 


Results from meta-analyses and benefit-cost modeling are then used to classify programs as 
evidence-based, research-based, or promising, based on the definitions in state law shown 
below. 

Legislative Definitions of Evidence-Based, Research-Based, and Promising Practices 
Evidence-based practice 

A program or practice that has been tested in heterogeneous or intended populations with multiple 
randomized, or statistically controlled evaluations, or both; or one large multiple site randomized, or 
statistically controlled evaluation, or both, where the weight of the evidence from a systemic review 
demonstrates sustained improvements in at least one outcome. "Evidence-based" also means a program 
or practice that can be implemented with a set of procedures to allow successful replication in 
Washington and, when possible, is determined to be cost-beneficial. 

Research-based practice 

A program or practice that has been tested with a single randomized, or statistically controlled 
evaluation, or both, demonstrating sustained desirable outcomes; or where the weight of the evidence 
from a systemic review supports sustained outcomes as described in subsection (14) of this section but 
does not meet the full criteria for evidence-based. 

Promising practice 

A practice that, based on statistical analyses or a well-established theory of change, shows potential for 
meeting the evidence-based or research-based criteria, which may include the use of a program that Is 
evidence-based for outcomes other than those listed in subsection (14) of this section (defining "evidence- 
based"). 

RCW 71.24.025. 


To classify programs, the criteria in the statutory definitions are operationalized as follows: 

1) Weight of evidence . To meet the evidence-based definition, results from at least one 
random-effects meta-analysis of multiple evaluations or one large multiple-site evaluation 
must indicate the practice achieves the desired outcome(s) (p-value < 0.20). To meet the 
research-based definition, at least one single-site evaluation must indicate the practice 
achieves desired outcomes (p-value < 0.20). 

2) Benefit-cost . The statute defining evidence-based practices requires that, when possible, a 
benefit-cost analysis be conducted. Programs that achieve at least a 75% chance of a 
positive net present value meet the "cost beneficial" criterion.® 


^ To operationalize the benefit-cost criterion, net benefits must exceed costs at least 75% of the time. After considerable analysis, we 
found that a typical program that WSIPP has analyzed may produce benefits that exceed costs roughly 75% of the time with a p- 
value cut off of up to 0.20. Thus, we determined that programs with p-values < 0.20 on desired outcomes should be considered 
research-based to avoid classifying programs with desirable benefit-cost results as promising. 
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3) Heterogeneity . To be designated as evidence-based, a program must have been tested on a 

"heterogeneous" population. We operationalize heterogeneity in two ways. First, the proportion of 
program participants belonging to racial/ethnic minority groups must be greater than or equal to 
the proportion of minority children in Washington. From the 2010 Census, for children age 0-17 in 
Washington, 68% were White and 32% belonged to racial/ethnic minority groups.^ Thus, if the 
weighted average of program participants in the outcome evaluations of the program was at least 
32% racial/ethnic minority, then the program was considered to have been tested in a 
heterogeneous population. 

Second, the heterogeneity criterion can also be achieved if at least one of a program's outcome 
evaluations has been conducted on children in Washington and a subgroup analysis demonstrates 
the program is effective for racial/ethnic minorities (p-value < 0.20). 

To summarize, we begin with the pool of programs defined at the outset and review the research 
literature for studies meeting WSIPP's criteria for methodological rigor. Programs that have no studies 
are not analyzed further, and these programs are noted in the report. Programs are deemed to be 
promising if some research on the program suggests effectiveness even though the studies do not 
meet WSIPP's methodological criteria or if the program has a well-defined theory of change. For 
programs that do have studies that meet WSIPP's methodological criteria, we conduct a meta¬ 
analysis. If the meta-analysis indicates at least one effect on an outcome of interest according to the 
weight of evidence criterion, the program is eligible to be either research-based or evidence-based. 

To reach the top tier, a program must also meet heterogeneity and benefit-cost criteria. Many 
interventions produce effects on more than one type of outcome. In our evidence ratings and benefit- 
cost results, we include all relevant outcomes, not just those related to substance use or marijuana. 

WSIPP has clarified classifications for programs that produce null or poor results since the last 
inventory update. In prior inventories, there was a single category for programs producing "null or 
poor outcomes." Programs with null effects on outcomes (p-value > 0.20) were inconsistently 
categorized as either "null or poor" or as "promising." For the current inventory, WSIPP has defined 
two separate categories to distinguish between programs producing null results (no significant effect 
on desired outcomes) and those producing poor (undesirable) outcomes. If results from a random- 
effects meta-analysis of multiple evaluations or one large multiple-site evaluation are not statistically 
significant (p-value > 0.20) for relevant outcomes, the practice may be classified as "null." If results 
from a random-effects meta-analysis of multiple evaluations or one large multiple-site evaluation 
indicate that a practice produces undesirable effects (p-value < 0.20), the practice may be classified as 
producing "poor" outcomes. If there is sufficient evidence of desirable effects on some outcomes but 
undesirable effects on other outcomes, we note the mixed results next to the program rating. 

Results of our classifications are displayed at the end of this report and are also available on our 
website.^^ Further information on the individual programs contained in the inventory can also be 
found on our website.^^ 


“ United States Census Bureau, 2010. 

Darnell, A., Goodvin, R., del Moral, S., Hicks, C., Wanner, P., & Westley, E. (2018). Updated inventory of programs for the prevention and 
treatment of youth cannabis use. (Document Number 18-12-3201). Olympia: Washington State Institute for Public Policy. 

WSIPP. Benefit-cost results. 
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Updates to the Inventory as of December 2018 


Since the previous publication of this inventory, WSIPP has updated the benefit-cost results for all 
programs^^ and has updated the literature reviews and meta-analyses for ten topics. Exhibit 3 
provides an overview of programs for which we changed classifications and the reasons for 
classification changes. 

There are a variety of reasons the classification for a program may change in an inventory update. 
These reasons include new research evidence, removing studies from the set of included studies, 
updating statistical calculations, and/or updating program costs. In this update of the cannabis 
inventory, the introduction of the null classification of programs also resulted in classification changes 
for some programs. In other cases, classifications changed because we based the rating on a broader 
set of outcomes than previously. Results have also changed due to updates to WSIPP's benefit-cost 
model and analyses. In November 2018 WSIPP completed an update to its benefit-cost model that 
reflects ongoing improvements to inputs and calculations across a variety of policy areas. We revised 
benefit-cost analyses using WSIPP's updated model for all eligible programs on the inventory. 


WSIPP's meta-analytic and benefic-cost methods are described in detail in our Technical Documentation. 
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Exhibit 3 


Classification Changes from Last Update and Reasons for Change 


Program name 

Former 

Current 

Reason for classification 

classification 

classification 

change 

Athletes Training and Learning to 

Avoid Steroids (ATLAS) 

Promising 

Null 

New null designation 

Caring School Community (formerly 

Child Development Project) 

Promising 

Null 

New null designation 

Compliance checks for tobacco 

Promising 

Research-based 

Classification based on revised 

set of outcomes 

keepin' it REAL 

Promising 

Null 

New null designation 

Marijuana Education Initiative 

No rigorous 
evaluations 

Promising 

Included new research 

Multicomponent environmental 
interventions to prevent youth alcohol 

use 

Promising 

Research-based 

Classification based on revised 

set of outcomes 

Project SUCCESS 

Poor outcomes 

Null 

New null designation 

Raising Healthy Children 

Promising 

Null 

New null designation 

Strengthening Families for Parents and 
Youth 10-14 

Research-based 

Null 

Included new evidence, 
removed studies from analysis 

Teen Marijuana Check-Up (TMCU) 

Evidence-based 

Research-based 

Benefit-cost 


Two new programs were added from the previous version of this inventory: Adolescent 
Community Reinforcement Approach (A-CRA), classified as research-based, and Sources of 
Strength, which was found to have no rigorous evaluations. In addition, we split the formerly 
unified category of community-based mentoring into two discrete programs; both are classified 
as research-based. Finally, five programs are not rated in this inventory because we found no 
studies meeting criteria for meta-analysis. 

Life Skills Training (for high school students) 

Love and Logic 
Project Venture 

Red Cliff Wellness School Curriculum 
Sources of Strength 
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Limitations 


The benefit-cost analyses in this report reflect only those outcomes that were measured in the 
studies we reviewed and are "monetizable" with the current WSIPP benefit-cost model. 
"Monetizable" means that we can link the outcome to future economic consequences, such as 
labor market earnings, criminal justice involvement, or health care expenditures. At this time we 
are unable to monetize some relevant outcomes, such as attitudes towards drug use or 
intentions to use. 
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December 2018 

Updated Inventory of Programs for the Prevention and Treatment of Youth Cannabis Use 


Program/intervention 

Level of 

evidence 

Effective for 

cannabis 

Benefit-cost 

percentage 

Reason program does not meet suggested evidence-based criteria 

(see full definitions below) 

Percent 

minority 

Prevention 

Alcohol Literacy Challenge (for college students) 

® 


48% 

Benefit-cost/heterogeneity 

24% 

Alcohol Literacy Challenge (for high school students) 

P 


58% 

Single evaluation 

33% 

Athletes Training and Learning to Avoid Steroids (ATLAS) 

Null 



Weight of the evidence 

22% 

Brief intervention for youth in medical settings 

® 


41% 

Benefit-cost 

65% 

Caring School Community (formerly Child Development Project) 

Null 


61% 

Weight of the evidence 

47% 

Communities That Care 

• 


85% 


33% 

Compliance checks for alcohol 

® 



Heterogeneity 

25% 

Compliance checks for tobacco 

® 



Heterogeneity 

28% 

Coping Power Program 

® 


54% 

Benefit-cost 

80% 

Curriculum-Based Support Group (CBSG) 

P 



Single evaluation 

90% 

Familias Unidas 

® 


41% 

Benefit-cost 

100% 

Family Check-Up (also known as Positive Family Support) 

® 

V 

49% 

Benefit-cost 

61% 

Family Matters 

® 


73% 

Benefit-cost/heterogeneity 

22% 

Guiding Good Choices (formerly Preparing for the Drug Free Years) 

® 


51% 

Single evaluation 

1% 

InShape 

® 


47% 

Single evaluation 

28% 

keepin' it REAL 

Null 


61% 

Weight of the evidence 

83% 

LifeSkills Training 

® 


59% 

Benefit-cost 

38% 

Lions Quest Skills for Adolescence 

® 

V 

65% 

Benefit-cost 

74% 

Marijuana Education Initiative 

P 



No rigorous evaluation measuring outcome of interest 


Mentoring: Community-based 

Mentoring: Big Brothers Big Sisters Community-Based (taxpayer costs only) 

® 


41% 

Benefit-cost 

57% 

Mentoring: Community-based (taxpayer costs only) 

® 


66% 

Benefit-cost 

85% 

Multicomponent environmental interventions to prevent youth alcohol use 

® 


28% 

Benefit-cost/heterogeneity 

19% 

Multicomponent environmental interventions to prevent youth tobacco use 

® 


85% 

Heterogeneity 

21% 

Positive Action 

• 


95% 


57% 

Project ALERT 

® 


70% 

Benefit-cost/heterogeneity 

12% 


9 Evidence-based (•) Research-based P Promising Null Null outcomes See definitions and notes on page 11. 

Notes : 

^ At least one cannabis outcome with a meta-analytic effect size estimate demonstrating reduced cannabis use with a p-value < 0.20. 

Many interventions produce effects on more than one type of outcome. This is especially true for prevention programs which often target multiple issues. WSIPP analyzes all relevant outcomes, and the 
evidence rating and benefit-cost results for a given program are often based on a variety of different outcomes, such as school achievement, substance use, mental health, and crime. In the column to 
the right of the level of evidence, we denote with a check mark those programs that have evidence of effectiveness for cannabis use specifically (p < 0.20). In addition to the overall level of evidence for a 
program, it is important to consider the specific outcomes the program has achieved to determine suitability for a given application. Each program name in the table links to a results page where a table, 
"Meta-Analysis of Program Effects," lists all of the outcomes analyzed for each program. 


The classifications in this document are current as of December 2018. 

For the most up-to-date results, please visit the program's page on our website http://www.wsipp.wa.gov/BenefitCost 
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December 2018 

Updated Inventory of Programs for the Prevention and Treatment of Youth Cannabis Use 


Program/intervention 

Level of 

evidence 

Effective for 

cannabis 

Benefit-cost 

percentage 

Reason program does not meet suggested evidence-based criteria 

(see full definitions below) 

Percent 

minority 

Prevention (continued) I 

Project Northland 



70% 

Benefit-cost 

36% 

Project STAR 

® 

V 

67% 

Benefit-cost/heterogeneity 

5% 

Project SUCCESS 

Null 


43% 

Weight of the evidence 

38% 

Project Toward No Drug Abuse 

® 


56% 

Benefit-cost 

70% 

PROSPER 

® 

V 

55% 

Benefit-cost/heterogeneity 

15% 

Protecting You/Protecting Me 

P 



Single evaluation 

92% 

Raising Healthy Children 

Null 



Weight of the evidence 

18% 

School-based tobacco prevention programs (including Project Towards No Tobacco Use) 

• 


99% 


41% 

SPORT 

® 


70% 

Benefit-cost 

49% 

STARS (Start Taking Alcohol Risks Seriously) for Families 

P 



Single evaluation 

66% 

Strengthening Families for Parents and Youth 10-14 

Null 


58% 

Weight of the evidence 

19% 

Strong African American Families 

® 



Single evaluation 

100% 

Strong African American Families—Teen 

® 



Single evaluation 

100% 

Teen Intervene 

® 

V 

72% 

Benefit-cost/heterogeneity 

29% 

Treatment 

Adolescent Assertive Continuing Care (ACC) 

® 

V 

37% 

Benefit-cost/heterogeneity 

27% 

Adolescent Community Reinforcement Approach (A-CRA) 

® 



Single evaluation 

59% 

Functional Family Therapy (FFT) for adolescents with substance use disorder 

® 


35% 

Benefit-cost 

74% 

Multidimensional Family Therapy (MDFT) 

® 

V 

25% 

Benefit-cost 

87% 

Multidimensional Treatment Foster Care 

® 

V 

63% 

Benefit-cost/heterogeneity 

24% 

Multisystemic Therapy (MST) forjuveniles with substance use disorder 

® 

V 

52% 

Benefit-cost 

65% 

Teen Mariiuana Check-Uo (TMCU) 

® 

V 

48% 

Benefit-cost 

35% 


0 Evidence-based (•) Research-based P Promising Null Null outcomes See definitions and notes on page 11. 

Notes : 

^ At least one cannabis outcome with a meta-analytic effect size estimate demonstrating reduced cannabis use with a p-value < 0.20. 

Many interventions produce effects on more than one type of outcome. This is especially true for prevention programs which often target multiple issues. WSIPP analyzes all relevant outcomes, and the 
evidence rating and benefit-cost results for a given program are often based on a variety of different outcomes, such as school achievement, substance use, mental health, and crime. In the column to 
the right of the level of evidence, we denote with a check mark those programs that have evidence of effectiveness for cannabis use specifically (p < 0.20). In addition to the overall level of evidence for a 
program, it is important to consider the specific outcomes the program has achieved to determine suitability for a given application. Each program name in the table links to a results page where a table, 
"Meta-Analysis of Program Effects," lists all of the outcomes analyzed for each program. 


The classifications in this document are current as of December 2018. 

For the most up-to-date results, please visit the program's page on our website http://www.wsipp.wa.gov/BenefitCost 
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December 2018 

Updated Inventory of Programs for the Prevention and Treatment of Youth Cannabis Use 


Definitions and Notes: 


Level of Evidence: 


Evidence-based : 


Research-based : 


A program or practice that has been tested in heterogeneous or intended populations with multiple randomized and/or statistically controlled evaluations, or one large multiple-site 
randomized and/or statistically-controlled evaluation, where the weight of the evidence from a systematic review demonstrates sustained improvements in at least one outcome. 
Further, "evidence-based" means a program or practice that can be implemented with a set of procedures to allow successful replication in Washington and, when possible, has been 
determined to be cost-beneficial. 

A program or practice that has been tested with a single randomized and/or statistically-controlled evaluation demonstrating sustained desirable outcomes; or where the weight of 
the evidence from a systematic review supports sustained outcomes as identified in the term "evidence-based" in RCW (the above definition) but does not meet the full criteria for 
"evidence-based." 


Promising practice : 
Null outcome(s) : 


A program or practice that, based on statistical analyses or a well-established theory of change, shows potential for meeting the "evidence-based" or "research-based" criteria, which 
could include the use of a program that is evidence-based for outcomes other than the alternative use. 

If results from multiple evaluations or one large multiple-site evaluation indicate that a program has no significant effect on outcomes of interest (p-value > 0.20), a program is 
classified as producing "null outcomes." 


Reasons Programs May Not Meet Suggested Evidence-Based Criteria: 

Benefit-cost : The proposed definition of evidence-based practices requires that, when possible, a benefit-cost analysis be conducted. We use WSIPP's benefit-cost model to determine whether a 

program meets this criterion. Programs that do not have at least a 75% chance of a positive net present value do not meet the benefit-cost test. The WSIPP model uses Monte Carlo 
simulation to test the probability that benefits exceed costs. The 75% standard was deemed an appropriate measure of risk aversion. 

Heterogeneity : To be designated as evidence-based under current law or the proposed definition, a program must have been tested on a "heterogeneous" population. We operationalized 

heterogeneity in two ways. First, the proportion of minority program participants must be greater than or equal to the minority proportion of children under 18 in Washington 
State. From the 2010 Census, of all children in Washington, 68% were White and 32% minority. Thus, if the weighted average of program participants had at least 32% minorities 
then the program was considered to have been tested on a heterogeneous population. 

Second, the heterogeneity criterion can also be achieved if at least one of the studies has been conducted on children in Washington, and a subgroup analysis demonstrates the 
program is effective for minorities (p-value < 0.20). Programs passing the second test are marked with a Programs that do not meet either of these two criteria do not meet the 
heterogeneity definition. Programs whose evaluations do not meet either of these two criteria do not meet the heterogeneity definition. 

No rigorous evaluation measuring outcome of interest : The program has not yet been tested with a rigorous outcome evaluation. 

Single evaluation : The program does not meet the minimum standard of multiple evaluations or one large multiple-site evaluation contained in the current or proposed definitions. 

Weight of evidence : To meet the evidence-based definition, results from a random-effects meta-analysis (p-value < 0.20) of multiple evaluations or one large multiple-site evaluation must indicate the 

practice achieves the desired outcome(s). To meet the research-based definition, one single-site evaluation must indicate the practice achieves the desired outcomes (p-value < 0.20). 

Other Definitions: 


Benefit-cost percentage: 


Benefit-cost estimation is repeated many times to account for uncertainty in the model. This represents the percentage of repetitions producing overall benefits that exceed 
costs. Programs with a benefit-cost percentage of at least 75% are considered to meet the "cost-beneficial" criterion in the "evidence-based" definition above. 
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For further information, contact: 

Adam Darnell at 360.664.9074, adam.darnell(a)wsipp.wa.gov Document No. 18-12-3201 

Washington State Institute for Public Policy 

The Washington State Legislature created the Washington State Institute for Public Policy in 1983. A Board of Directors—representing the 
legislature, the governor, and public universities—governs WSIPP and guides the developnnent of all activities. WSIPP's nnission is to carry out 
practical research, at legislative direction, on issues of importance to Washington State. 





